Skip to content

Addressing issue #1481 - #1499

Merged
VisLab merged 2 commits into
hed-standard:mainfrom
VisLab:address_issues
Nov 9, 2026
Merged

VisLab merged 2 commits into
hed-standard:mainfrom
VisLab:address_issues

Conversation

@VisLab

@VisLab VisLab commented Nov 9, 2026

Copy link
Copy Markdown
Member

No description provided.

Assembled validation of a timeline file parsed every row, discarded
the rows with an onset, and rechecked merged rows. Fixes hed-standard#1481.

- validate_assembled assembles once and skips a row with a numeric
  onset by row label before parsing. A row whose onset is n/a was
  tested by position in the sorted split frame, so it was skipped
  unless it was last; it is now checked wherever it is.
- Delete _recheck_duplicates, _is_within_tolerance and ONSET_TOLERANCE:
  a repeat in merged rows was reported twice, and the 1e-9 merge
  tolerance in df_util is now the only one.
- A bad Delay group is removed from a fresh parse, so the issue's HED
  string context keeps it.
- needs_sorting ignores n/a onsets (no ONSETS_UNORDERED for them);
  EventManager keeps requiring every onset to be numeric.
- combine_dataframe and _filter_by_index_list work on lists.
- 20,000 rows: 4.29 s -> 1.01 s with onset, 2.30 s -> 1.05 s without.
- Five regression tests; four tests no longer expect ONSETS_UNORDERED
  for n/a; row char_index now counts in the ", "-joined string.
Moves spec_tests/hed-tests from 568ac2b to b0bd646, the merge of
hed-tests PR hed-standard#98, which adds the TAG_EXPRESSION_REPEATED case
tag-expression-repeated-na-onset-row for hed-standard#1481.
@VisLab
VisLab requested a balanced review from Copilot November 9, 2026 14:31
@VisLab VisLab added bug Something isn't working enhancement New feature or request efficiency required-update labels Nov 9, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues requiring resolution.

No open issues 🎉

What changed in this PR

Addresses #1481 by reducing assembly overhead in spreadsheet validation, improving validation of large event tables used by downstream tools such as ndx-hed.

Changes:

  • Reuses assembled row strings, avoids redundant parsing and duplicate checks, and reduces pandas overhead.
  • Validates rows with n/a onsets consistently while preserving EventManager's numeric-onset requirement.
  • Preserves Delay error context and adds regression tests for merging, onset ordering, and error positions.
File Description
tests/​validator/​test_spreadsheet_validator.py Tests merged rows, missing onsets, merge tolerance, and Delay context.
tests/​validator/​test_hed_validator.py Updates expected character positions in assembled strings.
tests/​models/​test_tabular_input.py Verifies missing onsets do not trigger ordering warnings.
hed/​validator/​spreadsheet_validator.py Reuses assembly, removes redundant checks, and preserves error context.
hed/​tools/​analysis/​event_manager.py Explicitly requires numeric, non-decreasing onsets.
hed/​models/​df_util.py Uses list-based joins instead of positional pandas writes.
hed/​models/​base_input.py Streamlines row assembly and excludes missing onsets from ordering checks.
CHANGELOG.md Documents performance improvements and validation changes.

Review effort: Balanced

@VisLab
VisLab merged commit 9085ddd into hed-standard:main Nov 9, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working efficiency enhancement New feature or request required-update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants